1. Statement of the Technical Field
The present invention relates to computer communications network security and performance monitoring and more particularly to an intrusion detection system.
2. Description of the Related Art
Internet security has increasingly become the focus of both corporate and home computer users who participate in globally accessible computer networks. In particular, with the availability of broadband Internet access, even within smaller computer communication networks, most network attached computing devices enjoy continuous access to the Internet. Notwithstanding, continuous, high-speed access is not without its price. Specifically, those computers and computer networks which heretofore had remained disconnected from the security risks of the Internet now have become the primary target of malicious Internet hackers, crackers and script kiddies (relatively unskilled hackers), collectively referred to herein as “unauthorized intruders”.
Notably, many such unauthorized intruders continuously scan the Internet for Internet Protocol (IP) addresses and ports of vulnerable computers communicatively linked to the Internet. At the minimum, those vulnerable computers can experience nuisance damage such as accessed, deleted or modified files or defaced Web pages. Yet, at the other extreme, for the unsuspecting end-user, their computer can become the launching pad for more malicious attacks which can cripple whole segments of the Internet.
To combat the enhanced threat of unauthorized intruders, information technologists have liberally deployed firewall technology about the enterprise, at least to limit the source and channel of packet flow to and from the enterprise. Still, while firewall usage can limit the number of ports through which an unauthorized intruder can access an internal portion of a network, the firewall in of itself can be comprised by an unauthorized intruder. Popular examples include denial of service attacks and SYN flood attacks. Thus, while firewall usage can improve the security of a network, deploying firewall technology alone cannot completely secure the network.
To fill the security gap left open by firewall usage, information technologists incorporate intrusion detection system (IDS) technology within the enterprise. IDS technology can detect network intrusions dynamically as they occur or post-mortem after the intrusion has occurred. A typical dynamic network IDS, for instance the IDS disclosed in United States Patent Application Publication No. 2002/0035683 A1 to Kaashoek et al. for ARCHITECTURE TO THWART DENIAL OF SERVICE ATTACKS, can include a monitoring component able to capture network packets as the packets pass through the IDS, an inference component for determining whether the captured traffic indicates any malicious activity or usage, and a response component able to react appropriately to the detection of a malicious intrusion. While the response can include the generation and transmission of a simple e-mail message to a system administrator, the response also can include more complex actions, for instance temporarily blocking traffic flowing from an offenders Internet protocol (IP) address.
Conventional IDS technology can incorporate a variety of methodologies for determining within the inference component whether malicious activity has occurred or is occurring. Referred to as “detection methodologies”, examples can include simple pattern matching, stateful pattern matching, protocol decode-based signatures, heuristic-based signatures, and anomaly detection. Pattern matching is based upon inspecting traffic to identify a fixed sequence of bytes in a single packet. The fixed sequence of bytes, referred to in the art as a “signature”, when identified within inspected traffic, can trigger an alarm. U.S. Pat. No. 6,279,113 B1 to Vaidya for DYNAMIC SIGNATURE INSPECTION-BASED NETWORK INTRUSION DETECTION illustrates an exemplary use of pattern matching technology.
Still, pattern matching is considered to be the most primitive of the detection methodologies employed in a typical IDS. In that regard, pattern matching can fail where the hack attack differs only slightly from the stored signatures leading to what is known as “false negatives”—the failure to detect an attack after having inspecting traffic associated with an attack. Stateful pattern matching is an enhanced, more mature version of simple pattern matching based upon the notion that a stream of network traffic includes more than mere stand-alone packets. In consequence, pattern matching ought to be applied in the context of a stream of packets. To place pattern matching within the context of a stream of packets, the stateful pattern matching methodology considers the arrival order of packets in a stream and applies pattern matching to packets in the stream. Still, like simple pattern matching, stateful pattern matching can fail where the pattern of an attack differs only slightly from the stored signatures, again leading to false negatives.
Protocol decode-based analysis has been considered to be an intelligent extension to stateful pattern matching. In protocol decode-based analysis, traffic first is decoded in real-time according to a specified protocol such as HTTP in order to identify the pertinent fields of the protocol. Once the fields of the traffic specified by the protocol have been decoded, pattern matching can be applied to the decoded fields. U.S. Pat. No. 6,301,668 B1 to Gleichauf et al. for METHOD AND SYSTEM FOR ADAPTIVE NETWORK SECURITY USING NETWORK VULNERABILITY ASSESSMENT illustrates one such application of a protocol decode-based analysis.
As will be apparent from a review of the '668 system, the protocol decode-based analysis can limit the number of false alarms, or “false positives”, encountered during the matching process because much of the matched elements are placed into context through the decoding process. Still, the false positive rate of the protocol decode-based analysis is largely dependent on the accuracy of the publicly-specified protocol definition. Also, the success of the protocol decode-based analysis relies directly upon the freshness of the patterns used to identify unauthorized intrusions.
Unlike intrusion detection techniques which rely directly upon pattern matching, a heuristic-based analysis employs algorithmic logic upon which intrusion detection signatures can be based. Typically, the algorithmic logic can analyze traffic patterns in order to match a particular traffic pattern with a known “signature”. For instance, the probing of a network device can be detected where many unique ports are accessed over a limited period of time. Moreover, the type of packets touching the unique ports further can indicate whether an unauthorized intrusion is unfolding. Of course, any heuristic-based analysis can report false positives where a pattern of legitimate access to a network device satisfies the algorithmic logic. Hence, the use of a heuristic-based analysis requires extensive and frequent tuning to limit such false positives.
Similar to the heuristic-based analysis, in an anomaly-based analysis, traffic can be dynamically inspected as the traffic passes through the IDS. In an anomaly-based analysis, however, traffic patterns can be analyzed to detect anomalous behavior. Specifically, in an anomaly-based analysis, first a normal state is defined. Subsequently, traffic patterns which deviate from the normal state are labeled as unauthorized intrusions. Notably, though some anomaly-based analysis are configured to adapt the definition of normal state to traffic patterns as they unfold, none have been able to properly avoid the classification of some abnormal behavior as normal behavior. Moreover, no one conventional anomaly-based analysis has been able to distinguish anomalous behavior from permissible deviations from the normal state.
Nevertheless, IDS technology heretofore has been unable to provide a comprehensive method for detecting unauthorized intrusions while minimizing false positives. Specifically, static methods of detection such as pattern matching and its derivatives standing alone can be defeated by a sophisticated intruder with relative ease. Likewise, dynamic methods of detection such as those based upon heuristics and anomaly detection are limited to the extent that the methods can be configured improperly or ineffectively.
In addressing the deficiencies of the foregoing IDS methodologies, several IDS technologies incorporate a hybrid combination of static signature based pattern matching algorithms and dynamic anomaly detection algorithms. As an example, U.S. Pat. No. 6,321,338 to Porras et al. for NETWORK SURVEILLANCE discloses a method of network surveillance in which one or more analysis engines can perform both signature analysis and a statistical profiling of recorded network events. As discussed in column 4, lines 61 through 67 of the '338 specification, the event stream can be derived from a variety of sources, specifically the payload of a TCP/IP network packet or data contained in an analysis report.
Nevertheless, in view of the substantial processing resources required to reduce network traffic flowing across multiple network nodes, the '338 system processes only “events” detected and disseminated by a group of distributed monitoring components. That is to say, the '338 system does not process all network traffic flowing through the IDS with which the traffic can be analyzed to identify an unauthorized intrusion. Instead, the '338 system performs a tiered analysis of suspicious events in order to reduce the resource overhead which otherwise would be associated with a more thorough analysis.
More importantly, as the '338 system undertakes a fundamental analysis only of “event data” as stated in column 5, lines 34-35, the '338 system does not analyze traffic at a granular enough level to apply sophisticated statistical analyses. In particular, in the field of network analysis it is known to extract data from each individual packet field in a network packet in order to troubleshoot traffic flow in a network. U.S. Pat. No. 5,787,253 to McCreery et al. for APPARATUS AND METHOD OF ANALYZING INTERNET ACTIVITY describes such a device. Yet, in conventional IDS technology such as that described in the '338 system, the individual fields of a network traffic packet are never analyzed. In fact, in the '338 system, only the payload of an errant packet is extracted for analysis.
Ideally, to undertake the effective statistical analysis which is required to minimize false positives in the application of an anomaly based detection scheme, a maximum amount of data samples of exceptional granularity will be required. Thus, in the context of an IDS, it would be preferable to analyze each packet flowing across the IDS. Yet, to process each packet in-line would require an unreasonable share of processing resources. Moreover, to process each packet in batch would require substantial fixed storage and an unusually thorough analysis scheme not available through ordinary anomaly based detection schemes.
As an example, the IDS taught in U.S. Pat. No. 6,282,546 to Gleichauf et al. for SYSTEM AND METHOD FOR REAL-TIME INSERTION OF DATA INTO A MULTI-DIMENSIONAL DATABASE FOR NETWORK INSTRUCTION DETECTION AND VULNERABILITY ASSESSMENT employs the batch processing of real-time acquired data to detect an unauthorized intrusion. Yet, the '546 system performs a limited analysis only upon a limited set of scalar meta-data such as time, address space, and event type. Moreover, the '546 system performs an analysis only upon a limited subset of all traffic passing through the IDS—namely data already associated with an event such as an attack. The '546 system, then, does not analyze any volume of network traffic prior to the detection of an event. Thus, the '546 system like other conventional IDS implementations, cannot achieve a high level of intrusion detection while minimizing false positives.